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Abstract 

Stationary iterative methods with a symmetric splitting matrix are performed as inner- 
iteration preconditioning for Krylov subspace methods. We give a necessary and sufficient 
condition such that the inner-iteration preconditioning matrix is definite, and show that 
conjugate gradient (CG) method preconditioned by the inner iterations determines a solu¬ 
tion of symmetric and positive semidefinite linear systems, and the minimal residual (MR) 
method preconditioned by the inner iterations determines a solution of symmetric linear 
systems including the singular case. These results are applied to the CG and MR-type 
methods such as the CGLS, LSMR, and CGNE methods preconditioned by inner itera¬ 
tions, and thus justify using these methods for solving least squares and minimum-norm 
solution problems whose coefficient matrices are not necessarily of full rank. Thus, we 
complement the theory of these methods presented in [K. Morikuni and K. Hayami, SIAM 
J. Matrix Appl. Anal., 34 (2013), pp. 1-22], [K. Morikuni and K. Hayami, SIAM J. Matrix 
Appl. Anal., 36 (2015), pp. 225-250], and give bounds for these methods. 

Keywords: Rank-deficient least squares problems, Preconditioning, Krylov subspace 
methods, Symmetric singular linear systems. 
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I Introduction. 

First, consider solving symmetric linear systems of equation 

Ax = b, (1.1) 

where A = A T £ M nxri is not necessarily nonsingular and b is in the range space of A, 1Z(A). 

In the symmetric and positive definite (SPD) case, i.e., v T Av > 0 for all v y 0, the 
conjugate gradient (CG) method [23] has been used. In the symmetric and positive semidefinite 
(SPSD) case, i.e., v T Av > 0 for all v £ M n , for b £ TZ(A), CG with the initial iterate 
xo € M n determines the kth iterate x^ £ xo + /Cfc(A,ro) that minimizes the A-seminorm 

II e Jfc 11A = I|x fe — x*|| A , where r 0 = b - Ax 0 , K k { A,r 0 ) = span{r 0 , Ar 0 ,..., A fe_1 r 0 } is the 
Krylov subspace of order k, ||e||A = \J (e, e)A = v (Ae, e) is the semi inner product associated 
with A SPSD, and 

x* = A # b + (I — A # A)x 0 , (1.2) 
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is the group inverse of A, which satisfies 


AA# = I if A is nonsingular, 
A 2 A# = A if A is singular, 


AA # = A # A, A # AA # = A # , 


(see [41, Theorem 3.2]), and I is the identity matrix. Note that since index(A) < 1 from 
A = A t , A# is equal to the Drazin inverse of A, where index(A) = min{d £ No | rank A c/ = 
rankA (/+1 } and A 0 = I. CG determines the solution x * of (1.1) for all b £ 711(A) and for 
all xo € BN, and determines the minimum-norm solution A^b of (1.1) for all b £ 71(A) and 
for all xo £ 71(A). See [ 5], [20]. An error bound of CG is given by ||||a < 2[(a/ K 2 (A) — 
1)/(\/k 2(A) + l)] fc ||eo||A, e.g., [2 ], where K 2 (A) = |jA|| 2 1|A^|| 2 - Hence, the convergence is 
expected to be fast as ^(A) is small. 

In the indefinite case, the minimal residual (MR) method has been used. Particular imple¬ 
mentations of MR were presented in [39], [34], [1 ]. For b £ M n , MR with xo £ M n determines 
the kth iterate x^ £ xo + /Cfc(A,ro) that minimizes ||r^.|| 2 - MR determines a solution of least 
squares problems min xg R™ ||b —Ax ||2 for all b £ M and for all xo £ R”, determines the solution 
of the form (1.2) for all b £ 71(A) and for all xg £ R”, and determines the minimum-norm 
solution of (1.1) for all b £ 1Z(A) and for all xo € 711(A). These arguments are given by 
the symmetric case of the generalized minimal residual method [6], [19]. Similarly, a residual 

bound of MR is given by ||rfe ||2 < ||roH 2 with e k = min pe p fe max Ae(T (A) |p(A)|, where P fc is 

p(o)=i 

the set of all polynomials of degree not exceeding k and cr(A) is the spectrum of A [4, Theorem 
1]. See [17], [23], [39], [34], [40], [42], [13] for other Krylov subspace methods for symmetric 
linear systems. 

For accelerating the convergence of CG and MR, consider using preconditioning. See [ ] 

for the preconditioned CG method in the singular case. Several steps of stationary iterative 
methods serve as preconditioning for Krylov subspace methods, which may be considered as 
inner iterations [30]. We consider using stationary iterative methods with a symmetric split¬ 
ting matrix as inner-iteration preconditioning for CG and MR. To show that these methods 
determine a solution of symmetric and indefinite linear systems including the singular case, we 
give a necessary and sufficient condition such that the inner-iteration preconditioning matrix 
is definite and bounds for these methods. The condition is satisfied by using the Richard¬ 
son, Jacobi overrelaxation (JOR), and symmetric successive overrelaxation (SSOR) methods 
[35], [24], [3 ] whose relaxation parameters are within certain intervals. Thus, we improve the 
theories in the SPD case [12], [1] for a general symmetric case. Also, inner-iteration precon¬ 
ditioning is a tractable extension of the splitting preconditioning [36, Section 10.2], which can 
be regarded as one step application of the inner iterations. 

These methods have a potential to determine a solution of the normal equations 


A J Ax = A J b, 

(1.3) 

equivalently least squares problems 


min 5 — Ax 2 , 

(1.4) 

xGR 71 

where A £ R mxn is not necessarily of full rank and b £ R m 

is not necessarily in 1Z(A), us- 


ing efficient implementations such as the CGLS, LSQR, and LSMR methods [23], [33], [16]. 
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The normal equations (1.3) form an SPSD linear system. For solving (1.4), the (precondi¬ 
tioned) CGLS and LSQR methods have been used, which both are mathematically equivalent 
to (preconditioned) CG applied to (1.3). Another option for solving (1.4) is to use the (pre¬ 
conditioned) LSMR method [16], which is mathematically equivalent to (preconditioned) MR 
applied to (1.3). 

On the other hand, consider solving minimum-norm solution problems 

min ||a;|| 2 , subject to Ax = 6, b € 7Z(A). (1.5) 

The solution of (1.5) is the pseudo-inverse solution of Ax = 6, b e 11(A). The problem (1.5) 
is equivalent to the normal equations of the second kind 

x = A J u, subject to AA J u = b, b£7Z(A). (1.6) 

Note that the constraint of (1.6) is an SPSD linear system. For solving (1.5), the (precondi¬ 
tioned) CGNE method [10] has been used, which is mathematically equivalent to (precondi¬ 
tioned) CG applied to the constraint of (1.6). Another option for solving (1.5) is to use the 
(preconditioned) MRNE method [31], which is mathematically equivalent to (preconditioned) 
MR applied to the constraint of (1.6). 

We apply the above mentioned result for symmetric linear systems to CGLS, LSQR, LSMR, 
CGNE, and MRNE preconditioned by inner iterations, and thus justify using these methods 
particularly for rank-deficient least squares problems. 

CGLS and CGNE preconditioned by one step of SSOR-type methods were proposed in 
[5]. These methods were generalized to multistep versions in [30], [31]. In this paper, we 
complement the theory for the inner-iteration preconditioning for the CG and MR-type meth¬ 
ods including the rank-deficient case. These methods have an advantage concerning memory 
requirement compared to the right- and left-preconditioned generalized minimal residual meth¬ 
ods for least squares problems [21], [30], [3 ]. Numerical experiment results of these methods 
preconditioned by JOR and SSOR-type inner iterations were given in [3 ], [30]. 

The rest of the paper is organized as follows. In Section 2, we give conditions such that 
CG and MR preconditioned by inner iterations determine a solution of linear systems, and 
give bounds of these methods and conditions for specific stationary iterative methods that 
satisfy the condition. In Sections 3 and 4, we apply these results to CGLS, LSQR, and LSMR 
preconditioned by inner iterations for solving least squares problems and CGNE and MRNE 
preconditioned by inner iterations for solving minimum-norm solution problems, respectively. 
In Section 5, we conclude the paper. 

2 Preconditioning for symmetric linear systems. 

Consider solving symmetric linear systems (1.1). Let P = P T be a preconditioning matrix 
for (1.1). If P is SPD, then the linear system (1.1) is equivalent to the preconditioned one 
P _1 Ax = P -1 b, or 

P - ^AP = P ^b, x = P^^y (2.1) 

for all b € 1Z(A), where P 2 is the square root of P. If A = P~ 2 AP“ 2 , x = Pax, and 
b = P~zb, (2.1) becomes Ax = b. For b € 1Z(A), CG applied to (2.1) (PCG) determines 
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Xfc G xo + KLk {P X A,P 1 ro) that minimizes ||e*|| a j equivalently CG applied to P x Ax = 
P _1 b with the P-inner product does this, where e k = x. k — x*, 

x, = P'^A # b + P'MI- AA # )x 0 (2.2) 

(cf. (1.2)). For b G 7Z(A), MR applied to (2.1) (PMR) determines x^ G xo+/Cfc(P _1 A, P _1 ro) 
that minimizes ||ffcII 2 5 equivalently MR applied to P x Ax = P _1 b with the P-inner product 
does this, where r k = b — Ax*,. On the other hand, if P is symmetric and negative definite 
(SND), i.e., v t Av < 0 for all v/0, then the linear system Ax = b is equivalent to the pre¬ 
conditioned one (—P) _1 Ax = (— P)~ 1 b, or (— P ) - 2 A(— P) _ 2 y = (—P)~ 2 b, x = (— P) _ 2 y 
for all b G 7Z(A). Without loss of generality, we restrict ourselves to the case where the pre¬ 
conditioning matrix is SPD for simplicity hereafter. Even when the preconditioning matrix is 
SND, the arguments below hold by changing the sign. Thus, we obtain the following. 

Lemma 2.1. Assume that A is SPSD and P is SPD. Then, PCG determines a solution of 
Ax = b for all b G 7£(A) and for all xo G M n . The solution is of the form (2.2). 

Lemma 2.2. Assume thet. A = A T and P is SPD. Then, PMR determines a solution of 
Ax = b for all b G 7Z(A) and for all xo G M n . The solution is of the form (2.2). 

We note that PCG and PMR do not necessarily determine the minimum-norm solution 
A#b. Under the assumptions in Lemmas 2.1 and 2.2, PCG and PMR respectively determine 
the weighted minimum-norm solution argmin ||P — 2 a?|| 2 , subject to Ax = b for all b G 72.(A) 
and for all xo G 7£(P _1 A). 

2.1 Inner-iterations preconditioned methods. 

Consider using t steps of a stationary iterative method as inner-iteration preconditioning for 
CG for SPSD linear systems. Let be the preconditioning matrix of t inner iterations. An 
algorithm of this method is given as follows [3] (see [ ] for efficient implementations). 


Algorithm 2.1. CG method preconditioned by l inner iterations 

1. Let xo G M n be the initial iterate and ro = b — Axo- 

2. Apply t steps of a stationary iterative method to Az = ro to obtain zo = Po = C^ro- 

3. For k = 0,1,2,... until convergence, Do. 

4. a k = (r k , z*.)/(Ap*;, pfc), x fc+1 = x k + a k p k , r k+1 = r k - a k Ap k 

5. Apply t steps of a stationary iterative method to Az = r k+ \ to obtain z. k +\ = C^r^+i. 

6. /3fc — (rfc-)-i, )/(r^, Zfc), p^._|_i — z k +i T (d k p k 

7. EndDo 


We give an expression for the preconditioned matrix for CG with i inner iterations. Con¬ 
sider the stationary iterative method applied to Az = in lines 1 and 5. We call A = M—N a 
splitting of A and assume that M is nonsingular. Denote the iteration matrix by H = M -1 N. 
Assume that the initial iterate is z G Af(H), e.g., z(°) = 0. Then, the Ah iterate of the 
stationary iterative method is z-^ = Hz^ -1 ) + M _1 rfc = Z)i=o H*M _1 rfc,^ G N. Hence, the 
inner-iteration preconditioning matrix is 0 Therefore, the preconditioned 

matrix is C^A = H*(I — H) = I — H f . Conditions such that is dehnite will be 

given in Section 2.2. Under the conditions, one can set P -1 = and the arguments in 
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Lemma 2.1 follow. The inner-iteration preconditioning can be considered as the polynomial 
preconditioning using the truncated Neuman series expansion of M _1 A [9] (see also [12], [1 l], 
[32], [2], [7], [15], [8], [29] and references therein). 

On the other hand, consider the case of MR. Its algorithm is given as follows (cf. [ ]). 


Algorithm 2.2. MINRES method preconditioned by I inner iterations 

1. Let xo € M n be the initial iterate and ro = b — Axo 

2. Apply I steps of a stationary iterative method to Az = ro to obtain zo = C^ro- 

3. v 0 = 0, j3 = (r 0 , z 0 ) 2 , vi = r 0 //3, ui = z 0 /P, gi = 1, c_i = 1, 
s_i = 0, c 0 = 1, s 0 = 0 

4. For A: = 1,2,... until convergence, Do. 

5. w = Au k - /3fc_iv fe _i, a = (w, u k ), w k = w - a k v k 

6. Apply I steps of a stationary iterative method to Az = w k to obtain z k+ \ = C^w k . 

7. f3 k = (w fc , z fc+ i)3, Vfc+1 = w k /(3 k , u fc+ i = z k+l /p k 

8 . e k = s k ~ 2 ^k-i, $k = c k -2C k -i/3 k -i + s k -\a k 

9 ' ik = ~C k -2S k -lPk-l + Cfc-lOfc, p k = (Pi + 7fc 2 ) f 

10. c k — Tfc/ Pk: &k — Pk/Pk: — ^k7 k T s k (P k , g k — c k g k , g k ~\~ i — s k g k 

11- &k—l — (Ufc S k S k —2 3 )) d k — 1 — PSk: X k — X k -.\ + d k — \ Sfc_l 

12. EndDo 


The stationary iterative method applied to Az = ro and Az = in lines 2 and 5, 
respectively, gives the same preconditioning and preconditioned matrices as those in Algorithm 
2.1. Polynomial preconditioning for indefinite Hermitian systems is presented in [ ]. 


2.2 Definiteness of inner-iteration preconditioning matrices. 

In order to examine the definiteness of the inner-iteration preconditioning matrix C^, we 
extend [ , Lemma 1 and Theorem 1] to the general symmetric case A = A T . We denote 
A ~ B if A and B are similar and A = B if A and B are congruent. 


Lemma 2.3. Assume that A is SPD and AB is symmetric. Then, the eigenvalues of B are 
positive (negative) if and only if AB is positive (negative) definite. 

Proof. The lemma follows from AB ~ A 2 BA 2 = B. □ 


Lemma 2.4. Assume that AB is symmetric and B is SPD. Then, the eigenvalues of A are 
positive (negative) if and only if AB is positive (negative) definite. 

Proof. The lemma follows from AB ~ B 2 AB 2 = B. □ 


Lemma 2.5. If A is symmetric and definite (SD), i.e., either SPD or SND, and B = B T , 
then a(AB) C R. 

Proof. Assume that A is SPD. Then, we have ct(A 2 BA^ 2 ) c R, since it is symmetric. Hence, 
the eigenvalues of AB ~ A 2 BA 2 = B are real. On the other hand, assume that A is SND. 
Since -A is SPD, <r[(-A)(-B)] C R. □ 


Lemma 2.6. If A = A T and B is SD, then a(AB) C R. 
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Proof. Since the eigenvalues of (AB) t = b t a t are real from Lemma 2.5, we have cr(AB) C 

K. □ 

Lemma 2.7. If £ e N and <r(A) C R, then the eigenvalues of JfiZo A 1 are positive for all £ 
odd. 

Proof. If A is an eigenvalue of A not equal to 1, then the corresponding eigenvalue of YiZ o A* 
is (1 — A f )/(1 — A) > 0. If A = 1, then the corresponding eigenvalue of YiZo A* is £ > 0. Hence, 
the eigenvalues of YiZ o A* are positive for all £ odd. □ 

Now we show the main theorem and give a necessary and sufficient condition such that the 
inner-iteration preconditioning matrix is SD. 

Theorem 2.8 (cf. [1, Theorem 1]). Let A = A T be not necessarily nonsingular, £ € N, 
M = M t , N, H = M -1 N, CM = £-=d IBM- 1 be defined above. Then, the following hold. 

1. C W is symmetric. 

2. For £ odd, C W is positive ( negative ) definite if and only if M is positive (negative) 
definite. 

3. for £ even, is positive (negative) definite if and only i/M + N is positive ( negative ) 
definite. 

Proof for 1. Since N = N T , • • • M- 1 is symmetric. Hence, C^ is symmetric. □ 

Proof for 2. Assume that M is SPD (SND). Then, from Lemma 2.5, the eigenvalues of H = 
M- X (M — A) are real. Since M _1 is SPD (SND), noting Lemma 2.7, = Yn=o H*M _1 = 

(- Zto H i )(-M- 1 ) is SPD (SND). 

On the other hand, assume that is SPD (SND). Then, the eigenvalues of JL=o H* = 
CWm are real from Lemma 2.5, and positive from Lemma 2.7. Hence, from Lemma 2.3, 
M = (C^) -1 YaZq H* = (—C^) -1 (— Y^iZo H l ) is positive (negative) definite. □ 

Proof for 3. Since £ is even, we have 

MC W M = M + MH + MH 2 + MH 3 + • • • + MH^ 1 

= (M + MH) + (M + MH)H 2 + (M + MH)H 4 4-+ (M + MH)H f ~ 2 

= (M + N)(I + H 2 + H 4 + --- + H^ 2 ). (2.3) 

Assume that M + N is SPD (SND). Since G = E^ 2)/2 (H 2 ) 1 = (M + NJ^MC^M, 
we have <r(G) C R from Lemma 2.5, and A > 0 for all A € cr(G) from Lemma 2.7. Hence, 
MC^M = (M + N)G = —(M + N)(—G) = is positive (negative) definite. 

On the other hand, assume that = MC^M is SPD (SND). Then, from (2.3), M + N 
is nonsingular. Since (M + N) _1 is symmetric, the eigenvalues of G = (M + N) _1 MC^ f ^M 
are real from Lemma 2.6, and positive from Lemma 2.7. Hence, M + N = MC^MG -1 = 
(-MCWM)(-G- 4 ) is positive (negative) definite from Lemma 2.4. □ 

Letting A be positive definite in Theorem 2.8, we obtain [1, Theorem 1] as a corollary. 
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2.3 Convergence conditions. 

We give sufficient conditions such that CG and MR preconditioned by inner iterations deter¬ 
mine a solution of symmetric linear systems. 

Theorem 2.9. Assume that A = A T is not necessarily nonsingular and M = M T is nonsin¬ 
gular such that A = M — N. Then, CG preconditioned by l steps of the inner iterations 
defined above with M definite for £ odd and M + N definite for £ even, determines a solution 
of Ax = b with A SPSD for all b E 77(A) and for all xo E M n . 

Proof. From Theorem 2.8, is SD for all IeN, Theorem 2.1 applied to CG for C^ ) A T x = 
C^b with the inner product gives the theorem. □ 

Theorem 2.10. Under the same assumption in Theorem 2.9, MR preconditioned by £ steps 

of the inner iterations defined above with M definite for £ odd and M + N definite for £ 

even, determines a solution of Ax = b for all b E 77(A) and for all xo E M". 

Proof. From Theorem 2.8, is SD for all £ E N. Theorem 2.2 applied to MR for A T x = 
C^b the CM inner product gives the theorem. □ 

The solutions determined by these methods are given similarly to (2.2) with P _1 = 
Theorems 2.9 and 2.10 will be applied to CG and MR-type methods for least squares and 
minimum-norm solution problems in Sections 3 and 4. 

We note a relationship among definiteness, P-regularity, and semiconvergence. For a square 
matrix A, we say the splitting A = M — N is P-regular if M is nonsingular and M + N is 
positive definite, i.e., the symmetric part of M + N is SPD. Let A = M — N be P-regular 
for A symmetric, equivalently M + M T — A positive definite. Note that if M = M T , then 
M + M t — A = 2M — A = M + N. Then, H = M^ 1 N is semiconvergent, i.e., lim^oo H' 
exists, if and only if A is positive semidehnite [28, Theorem 2], Hence, for A indefinite, 
H = M _1 N is not semiconvergent even if A = M — N is P-regular. Therefore, from Theorem 
2.10, MR preconditioned by the inner iterations can determine a solution of Ax = b even if 
H is not semiconvergent, i.e., divergent. For exmaple, if A = diag(l,—1) = M — N, M = I, 
and N = diag(0, 2), then M and M + N are SPD but H = M _1 N = N is divergent. 

2.4 Convergence bounds. 

Consider convergence bounds of CG and MR preconditioned by inner iterations. For the 
definiteness of the preconditioning matrix, assume that C- l is SPD, or M and M + N are 
definite for £ both odd and even from Theorem 2.8. 

First, we focus on CG. Assume that A is SPSD or symmetric and negative semidehnite 
(SNSD), i.e., v t Av < 0 for all v E R n , and let H = M -1 N. From the proof of Theorem 
2.8, we have er(H) C R. Denote the pseudo spectral radius of H by J'(H) = max{|A| : A E 
(7 (H)\{ 1}} and the largest and smallest eigenvalues of H not equal to 1 by A max (H) and 
A m m(H), respectively. Since H is semiconvergent [28, Theorem 2], equivalently z/(H) < 1 and 
the eigenvalues of H equal to 1 are simple [22], we have 

(C {£) A) = H 1 - A max (H)]/[l - A min (H)] for £ odd, 

\ (1 — <5 £ )/[l — ^(H) £ ] for £ even, 
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where <5 is the eigenvalue with the smallest absolute value of H. If = K2 (cWA), then 
an error bound of CG preconditioned by £ inner iterations is given as ||e ^.||^4 < 2[(V — 
i)/(Vi^ + I)] fc || eg 11 - Thus, similar arguments in [1, Section 2.2] can be applied to the 
present SPSD (SNSD) case, defining the smallest eigenvalue of the iteration matrix by A r . In 
order to avoid repetition, we omit the detail. 

On the other hand, we give a bound of MR preconditioned by £ inner iterations. 


Theorem 2.11. If A is SPSD and H is semiconvergent, then the kth residual r*, of MR 
preconditioned by £ steps of the inner iterations define above satisfies 


|ffc ||2 < min 


n(U) H ,2 


d^T+i 


To II2 


(2.4) 


for all b E 7£(A) and for all xo E M n , where 2 r^. 

Proof. Theorem 2.10 ensures that MR preconditioned by the £ steps of the inner iterations 
determines a solution of Ax = b for all b E 7£(A) and for all xo E M n . From [ , Theorem 1], 
we have 


Tfc 2 = mm 
?>e p fe 
p(o)=i 


p( ACW)f 0 


< 


( min max |p(A)| llfolb- 

; { o)=\ a& (ac (<) ) ) 


Since the eigenvalues not equal to zero of AC ^ are in the circle with radius p(H Y < 1 with 
center at 1, [4, Theorems 2, 5] gives the bound i/(H) fc ^ of the first factor. Similarly to the error 
bound of CG using the condition number of the coefficient matrix, the residual bound of MR 
is obtained (cf. [36, Secion 6.11.3]). □ 

From Theorem 2.11, the convergence of MR preconditioned by inner iterations is expected 
to be fast as the spectral radius is small and/or the number of inner iterations are large. 

We compare the convergence of MR preconditioned by inner iterations with the stationary 
iterative method alone that is used as inner iterations for MR. If their (inner) iteration ma¬ 
trices are the same and semiconvergent, then the convergence of MR preconditioned by inner 
iterations is not worse in terms of the number of outer iterations vs. the residual norm. This 
is because the convergence factor of the stationary iterative method is H) fc , which is larger 
than the factors in (2.4). However, their computational costs of each iteration are not the 
same, the total costs required to attain a certain stopping criterion are easily comparable in 
theory. 

Since it is assumed in Theorem 2.11 that H is semiconvergent, which is weaker than 
that M + N is SD, Theorem 2.11 looses generality concerning the indefiniteness. We showed 
Theorem 2.11 for the application of MR to the normal equations, whose coefficient matrices 
are SPSD. 


2.5 Specific inner-iteration preconditioning methods. 

Theorem 2.8 gives insights for justifying the use of specific stationary iterative methods as 
inner-iteration preconditioning for CG and MR for solving SPSD and indefinite systems, re¬ 
spectively. 
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Let uj E M hereafter. The splitting matrix a; _1 I of A gives the Richardson method for 
(1.1) if uj 7 I 0. For odd £, the inner-iteration preconditioning matrix of the Richardson 
method is definite if uj 7 I 0. Let A = L + D + L T , where L is strictly lower triangular and D 
is diagonal. Then, the splitting matrix o; _1 D of A with D nonsingular gives JOR for (1.1) if 
w/0. For odd £, the inner-iteration preconditioning matrix of JOR is definite if uj 7 ^ 0 
and D is definite. For £ even, we show the following. Note that the splitting A = M — N gives 
M + N = 2M — A. 

Lemma 2.12. Let A = A T , B SPD and M = uj 1 B. Denote the largest eigenvalue of 
B 2 A.B 2 by A mav ■ Then, 2M — A is SPD if and only if uj E (0,2/A max ) for A max > 0, 
u 0 [2/A max , 0] for A max <0, or uj > 0 for A max = 0. 

Proof. Denote an eigenvalue of B" 2 AB" 2 by A. Then the corresponding eigenvalue of 2a;” 1 !— 
B 2 A.B 2 = 2a;" 1 B —A = 2M —A is 2 u ~ 1 — A. From 2a ;" 1 — A > 0 for all A E cj(B"2 AB"2) ; 
we have the intervals of a; for the positive definiteness of 2M — A. □ 

From Lemma 2.12 with the splitting matrices B = I and D SPD, we obtain the interval 
of the relaxation parameter uj for the definiteness of the Richardson and JOR inner-iteration 
preconditioning matrices for £ even. We omit the details to avoid redundancy. From 
Lemma 2.12, with an SPD splitting matrix not necessarily diagonal, we can generalized JOR. 

Next, consider SSOR applied to (1.1). Let (o; _1 D + L) be the splitting matrix of A for 
the forward sweep of SSOR and ( uj ^D-f L T ) be that of A for the backward sweep. 

Theorem 2.13. Assume that D is SPD. Then, the SSOR splitting matrix M = a;" 1 (2 — 
a;)" 1 (D + a;L)D~ 1 (D-|-a;L T ) of A is nonsingular if and only if uj / 0, 2. For l odd, the SSOR 
inner-iteration preconditioning matrix is SPD if and only if uj E (0, 2). Let p = A m i n [D "2 (L + 
L T )D 2 ]+ 1 and 

Ps = A max (D 2(L + L T )D 2)-|-2A max (D 2LD 1 L t D 2) + l. 

Then, for £ even, the SSOR inner-iteration preconditioning matrix is SPD if 


UJ 

E 

(0,(1 -VT 

- 2 [T)Ip) 

for 

p < 1/2, p 7^ 0. 

UJ 

E 

(0,1) 


for 

p = o, 

UJ 

E 

(0,2) 


for 

p > 1/2, 

UJ 

0 

[(1 + V^ 

2p s )/p s , 2] 

for 

Ps ^ 0 , 

UJ 

> 

2 


for 

Ps = 0 , 

UJ 

E 

(2, (i + >/r 

- 2p s )/p s ) 

for 

Ps E ( 0 , 1 / 2 ). 


Proof. Since a;" 1 (2 — a;)" 1 (D + a;L)D" 1 (D + a;L T ) = a;" 1 (2 — a;) -1 I, the SSOR splitting 
matrix of A is nonsingular if uj ^ 0, 2. For £ odd, from Theorem 2.8, the SSOR inner-iteration 
preconditioning matrix is SPD if and only if uj E (0,2). Next, let £ even. Assume uj ^ 0, 2. 
Noting 

2M — A = 2a; -1 (2 - a;)“ 1 (D + a;L)D" 1 (D + a;L T ) - A 

= a; _1 (2 - a;)" 1 [(a; 2 - 2 uj + 2)1 + a; 2 D"i(L + L T )D"5 + 2a; 2 D"iLD" 1 L T D"5], 
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let 


G(w) = ui(2 — w)D a(2M — A)D 2 

= (cj 2 — 2cj + 2)1 + cu 2 D A (L + L T )D A + 2 w 2 dAlD _1 L t dA. 

If v = v(w) is a real vector such that ||v|| 2 = 1 and G(cu)v = Av, then A E R is an eigenvalue 
of G(w). Note A > pui 2 — 2u + 2 and A < p s u 2 — 2cu + 2. Noting oj( 2 — u) > 0 for u S (0, 2), if 
the first three conditions in (2.5) hold, then the SSOR inner-iteration preconditioning matrix 
is SPD. Noting uj(2 — oj) < 0 for uj fL [0, 2], if the last three conditions in (2.5) hold, then the 
SSOR inner-iteration preconditioning matrix is SPD. □ 

Note that the SSOR iteration matrix of A SPSD with D SPD and u € (0, 2) is semicon- 
vergent [11, Theorem 14]. 

Similar arguments to the above for D SND and for the case where C is SND hold but 
we omit the details. 

3 Application to least squares problems. 

Consider solving linear least squares problems (1.4). We apply results in Section 2 to CGLS, 
LSQR, and LSMR preconditioned by inner iterations with A = A T A and b = A T b, and reveal 
the theory behind the inner-iteration preconditioning [30], [31] in the next subsection. 

3.1 Inner-iteration preconditioning. 

We give an expression for the preconditioned matrix for CGLS and LSQR with £ inner itera¬ 
tions, similarly to Section 2.1. Consider the stationary iterative method applied to A T Az = sj- 
in lines 2 and 6 in [31, Algorithm E.l]. Let M be nonsingular such that A J A = M — N. Denote 
the iteration matrix by H = M~ l N. Assume that the initial iterate is z ® = 0. Then, the ith. 
iterate of the stationary iterative method is +M _1 Sfc = JfiZ 0 H l Ai~ 1 sj i ,£ > 0 . 

Hence, the preconditioning matrix is C ^ qLPM - 1 . Therefore, the preconditioned 

matrix is C^A J A = £-“o H \I —H) = I -H e . 

See [ ] for a different formulation of CGLS preconditioned by the SSOR splitting. 

Next, consider LSMR preconditioned by inner iterations. The stationary iterative method 
applied to A J Az = v & in lines 3 and 7 in [31, Algorithm E.2] gives the same preconditioning 
and preconditioned matrices as above. Now we give conditions such that CGLS, LSQR, and 
LSMR preconditioned by inner iterations determine a least squares solution. 

Theorem 3.1. Let A € M mxn and, M = M J be nonsingular such that A J A = M — N. Then, 
CGLS, LSQR, and LSMR preconditioned by £ steps of the inner iterations defined above with 
M definite for £ odd and M + N definite for £ even, respectively, determine a solution of 
min^gRn ||6 — Ae ||2 for all b € R m and for all xq e R n . 

Proof. Since A r A is SPSD, the following hold from Theorem 2.8. For £ odd, is SPD 
(SND) if and only if M is SPD (SND). For £ even, is SPD (SND) if and only if M + N is 
SPD (SND). Hence, Theorems 2.10 and 2.10 complete the proof. □ 

Remark that this theorem holds whether A is of full-rank or rank-deficient, and whether 
A is overdetermined or underdeter mined, i.e., unconditionally with respect to A. We have 
bounds of these methods from Section 2.4. 
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3.2 Specific inner-iteration preconditioning methods. 

There are efficient implementations of inner-iteration preconditioning without explicitly form¬ 
ing AA J such as the Richardson-NE, Cimmiuo-NE, and NE-SSOR methods [ , Appendix D], 

which are mathematically equivalent to the Richardson method, JOR, and SSOR applied to 
the normal equations of the second kind, respectively. CGLS preconditioned by one step of 
NE-SSOR was considered in [5]. 

Let A t A = L + D-\-L J , where L is strictly lower triangular and D is diagonal. Assume that 
A has no zero columns. Then, D is SPD. If A = A J A and b = A T b in Section 2.5, then we 
obtain the intervals of the parameter values of Richardson-NE, Cimmino-NE, and NE-SSOR 
such that their inner-iteration preconditioning matrices are SD. Thus, from Theorem 3.1, 
CGLS, LSQR, and LSMR preconditioned by these inner iterations with relaxation parameters 
within the intervals determines a solution of min^gjRn ||b — Ace||2 for all b € M m and for all 
xq e R”. 

4 Application to minimum-norm solution problems. 

Consider solving minimum-norm solution problems (1.5). Results in Section 2 can also be 
applied to CGNE and MRNE preconditioned by the Richardson-NE, Cimmino-NE, and NE- 
SSOR inner iterations with A = AA T and b = b. CGNE preconditioned by one step of 
NE-SSOR was considered in [5]. We generalize this to a multistep version of NE-SSOR. 

4.1 Inner-iteration preconditioning. 

We give an expression for the inner-iteration preconditioning and preconditioned matrices for 
CGNE with £ inner iterations. Consider the stationary iterative method applied to AA T z = 
in lines 2 and 6 in [ , Algorithm E.3]. Let M be nonsingular such that AA T = M — N. Denote 

the iteration matrix by H = M~ 1 N. Assume that the initial iterate is = 0. Then, the ith. 
iterate of the stationary iterative method is = Hz + M~ l r k = YiJ) HlM 1 rkJ> 
0. Hence, the preconditioning and preconditioned matrices are = JT=o H l M~ l and 
C W AA J = IP (I -H) = I -H e , respectively. 

Next, consider MRNE preconditioned by inner iterations. The stationary iterative method 
applied to AA J u = c in lines 2 and 6 [31, Algorithm E.4] gives the same preconditioning and 
preconditioned matrices as above. Since AA T is SPSD, the following hold from Theorem 2.8. 
For l odd, C® is SPD (SND) if and only if M is SPD (SND). For £ even, C® is SPD (SND) 
if and only if M + N is SPD (SND). Thus, we obtain the following similarly to Theorem 3.1. 

Theorem 4.1. Let A € M mxn and M = M T be nonsingular such that AA J = M — N. Then, 
the CGNE and MRNE methods preconditioned by l steps of the inner iterations defined above 
with M definite for t odd and M + N definite for t even, respectively, determine a solution of 
Ax = b for all b e N(A) and for all xq e 1 Z(A). 

Remark that this theorem holds whether A is of full-rank or rank-deficient, and whether 
A is overdetermined or underdeter mined, i.e., unconditionally with respect to A. We have 
bounds of these methods from Section 2.4. 
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4.2 Specific inner-iteration preconditioning methods. 

Let AA t = L + D + L T , where L is strictly lower triangular and D is diagonal. Assume that A 
has no zero rows. Then, D is SPD. If A = AA r and b = b in Section 2.5, then we obtain the 
intervals of the parameter values of Richardson-NE, Cimmino-NE, and NE-SSOR such that 
their inner-iteration preconditioning matrices are SD. Thus, from Theorem 4.1, CGNE and 
MRNE preconditioned by these inner iterations with relaxation parameters within the intervals 
determines the pseudo-inverse solution of Ax = b for all b £ K(A) and for all Xq £ TZ(A t ). 

5 Conclusions. 

We considered applying stationary iterative methods with a symmetric splitting matrix as 
inner-iteration preconditioning to Krylov subspace methods. We gave a necessary and sufficient 
condition such that the inner-iteration preconditioning matrix is definite, and show that CG 
and MR preconditioned by the inner iterations determines a solution of symmetric linear 
systems including the singular case. Applying these results to CGLS, LSQR, LSMR, CGNE, 
and MRNE preconditioned by inner iterations, and we guaranteed using these methods for 
solving least squares and minimum-norm solution problems whose coefficient matrices are not 
necessarily of full rank. 
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